A screen capture tool lets a computer user record an image displayed on a visual display unit such as a computer monitor. The user might use the captured screen area (alternatively called a screen area, screen image, screen shot, screen frame, screen region, capture area, capture image, capture shot, etc.) in a help manual or report to show the results displayed on the display unit at a particular time.
FIG. 1a is a captured screen area (100) of a computer desktop environment according to the prior art. The captured screen area (100) shows the entire desktop, but could instead show only the window (130) or some other portion of the desktop. A cursor graphic (140) overlays the window (130), and several icon graphics (120, 122, 124) overlay the background (110).
For some applications, a user captures a series of screen areas to show how screen content changes. The user might use the series of captured screen areas within an instructional video for job training or remote instruction.
FIGS. 1b and 1c show captured screen areas (101, 102) following the captured screen area (100) of FIG. 1a in a series according to the prior art. Much of the screen content shown in FIGS. 1a-1c is identical. Screen content such as the background (110) and icon graphics (120, 122, 124) usually does not change from frame to frame. On the other hand, the cursor graphic (140) often changes position and shape as the user manipulates a mouse or other input device, and the contents of the window (130) often change as a user types, adds graphics, etc. FIG. 1b shows the cursor graphic (140) and the window (130) changing locations as the user drags the window (130) across the desktop, which in turn changes which portions of the background (110) are exposed. FIG. 1c shows the contents of the window (130) changing after typing by the user, while the cursor graphic (140) has disappeared.
When a series of screen areas is captured in quick succession (for example, 10 frames per second) or when a window displays slowly changing content, changes in screen content from frame to frame tend to be small. On the other hand, when screen capture is infrequent or when a window displays quickly changing content such as a video game or animation, changes from frame to frame tend to be more pronounced. Dramatic changes in screen content can also occur when windows or menus are opened, closed, moved, resized, etc.
The quality of a series of captured screen areas depends on several factors. Higher resolution and higher frame rate increase quality, but also increase performance costs. To understand how quality affects performance of a screen capture tool, it helps to understand how a computer represents and captures screen areas.
I. Computer Representation of Captured Screen Areas
A single rectangular captured screen area includes rows of picture elements [“pixels”] with color values. The resolution of the captured screen area depends on the number of pixels and the color depth. The number of pixels is conventionally expressed in terms of the dimensions of the rectangle, for example, 320×240 or 800×600. The color depth is conventionally expressed as a number of bits per pixel, for example, 1, 8, 16, 24 or 32, which affects the number of possible colors for an individual pixel. If the color depth is 8 bits, for example, there are 28=256 possible colors per pixel, which can be shades of gray or indices to a color palette that stores 256 different 24-bit colors in the captured screen area. A captured screen area represented by pixels and stored as a collection of bits, with each pixel having a color value, is an example of a bitmap.
The frame rate of a series of captured screen areas (i.e., the resolution in time) is conventionally expressed in terms of frames per second [“fps”]. Some conventional frame rates are 1, 2, 10, 15, 25, and 30 fps. A higher frame rate generally results in smoother playback of changing screen content.
Quality affects the number of bits needed to represent a series of captured screen areas, which in turn affects the costs of capturing, processing, storing, and transmitting the series. Table 1 shows the bit rates (bits per second) of several uncompressed series of captured screen areas of different quality.
Spatial ResolutionColor DepthFrame RateBit Rate(pixels hor × vert)(bits)(fps)(bits per second)320 × 240821,228,800320 × 2402423,686,400800 × 60024223,040,000800 × 6002410115,200,000Table 1: Bit Rates of Series of Captured Screen Areas of Different Quality.
The preceding examples generally illustrate representation of captured screen areas in computer systems. In practice a variety of different formats and conventions are used in various different representations of captured screen areas in different computer systems and stages of processing.
II. Display and Capture of Captured Screen Areas
Most computer systems include a display card, which stores information for output to a visual display unit. Common terms for display card include video card, graphics card, graphics output device, display adapter, video graphics adapter, etc.
On the display card, a frame buffer stores pixel information from which the display unit is refreshed. In addition to the frame buffer, the display card can include a graphics processor, graphics accelerator or other hardware to make display functions more efficient. A digital to analog converter converts digital information in the frame buffer to an analog form, and the analog information is transmitted to the display unit. Conventionally, screen content is refreshed pixel-by-pixel across a row of the display unit, the rows are refreshed row-by-row from top to bottom, and the process repeats such that the entire screen is refreshed 60 or more times per second. In one common scenario, a computer system loads display driver software for the display card into system memory. The computer system accesses various features of the display card through display driver software.
In a conventional screen capture operation, information is transferred from the display card frame buffer back to system memory of the computer system. The display driver and/or other layers of software in the computer system often facilitate such transfer by supporting a Bit Block Transfer [“BitBlt”] operation, which a screen capture tool can utilize. In general, in a BitBlt operation, the computer system transfers pixel information from a source (e.g., display card frame buffer) to a destination (e.g., system memory). Depending on implementation, the software application can specify parameters such as the source, the destination, and the coordinates and dimensions of a rectangle in the source or destination for which information should be retrieved.
III. Performance Costs of Screen Capture
High resolution, frame rate, and color depth tend to improve quality but also increase performance costs in terms of capture, processing, storage, and transmission. For screen capture applications, there are several performance bottlenecks.
Since a series of captured screen areas can have a very high bit rate, there can be performance bottlenecks at the points of storing the series or transmitting the series across a network. Compression of captured screen areas is often used to address these performance bottlenecks by decreasing bit rate (at times, at some cost to quality).
Another performance bottleneck occurs at the point of screen capture. BitBlt operations are a bottleneck during screen capture due to two factors: 1) the amount of data transferred from the display card frame buffer to the system memory and 2) color format conversions that may be required between the frame buffer and the system memory. For instance, some display cards store pixels in YUV color coordinates, while others use a non-planar RGB representation. A BitBlt operation typically converts the pixel information in the frame buffer to a standard representation such as planar RGB. FIG. 2 illustrates the bottleneck between system memory (210) and a display card frame buffer (220) of a computer system (200) during screen capture. While operations (212, 222) within system memory (210) or the display card frame buffer (220) are fast, the BitBlt operation (232) is slow.
The bottleneck between the display card frame buffer and system memory impedes screen capture at high resolutions and frame rates. As a result, a user may have to sacrifice spatial resolution, color depth, and/or frame rate to produce a series a captured screen areas with a screen capture tool.
Several attempts have been made in screen capture tools to address the bottleneck between the display card frame buffer and system memory. In one attempt, a screen capture tool performs a full screen capture when some developer-defined event (such as a user causing the rotation of an object displayed on the screen) triggers the screen capture. The rest of the time the screen capture tool captures cursor movement or nothing at all. While this helps eliminate screen capture operations in carefully controlled scenarios, it can be inflexible and inefficient. Screen content may change but not trigger capture events, so the changes do not show up in a series of captured screen areas. Moreover, a developer must define the capture events, which can require specialized knowledge and access to source code. Finally, the capture events trigger full captures of the screen areas, even if most of a captured screen area did not change.
In another attempt to address the bottleneck between the display card frame buffer and system memory, rather than performing full screen captures, a screen capture tool captures screen areas that change when a user moves a mouse, clicks a mouse button, or presses a key on the keyboard. This helps provide smoother capture of cursor movement and changes in a foreground window, but can completely miss other changes in screen content, such as a background animation or user interface updates. For instance, web pages often have user interface containers that update themselves without any user interaction.
While prior attempts to address the bottleneck between the display card frame buffer and system memory improve performance in some scenarios, they lack flexibility and are inefficient in many other scenarios.